Estimation of panoramic camera orientation relative to a vehicle coordinate frame

ABSTRACT

A system and method are presented for estimating the orientation of a panoramic camera mounted on a vehicle relative to the vehicle coordinate frame. An initial pose estimate of the vehicle is determined based on global positioning system data, inertial measurement unit data, and wheel odometry data of the vehicle. Image data from images captured by the camera is processed to obtain one or more tracks, each track including a sequence of matched feature points stemming from a same three-dimensional location. A correction parameter determined from the initial pose estimate and tracks can then be used to correct the orientations of the images captured by the camera. The correction parameter can be optimized by deriving a correction parameter for each of a multitude of distinct subsequences of one or more runs. Statistical analysis can be performed on the determined correction parameters to produce robust estimates.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 12/708,302, filed Feb. 18, 2010, which claims the benefit ofthe filing date of U.S. Provisional Patent Application No. 61/177,614,filed May 12, 2009, and U.S. Provisional Patent Application No.61/154,217, filed Feb. 20, 2009, both entitled “Estimation of PanoramicCamera Orientation Relative to a Vehicle Coordinate Frame,” the entiredisclosures of which are hereby incorporated herein by reference.

BACKGROUND

1. Field of the Invention

The present invention relates to camera-obtained imagery captured from amoving vehicle.

2. Related Art

A camera, such as a panoramic camera, can be mounted on a vehicle, suchas a car, truck, van, or any kind of vehicle, and used to capture imagesas the vehicle moves. A panoramic camera is a camera, typically a systemof one or multiple cameras, that is configured or arranged to capture apanoramic image (i.e., an image or view of an area in many directions,possibly every direction). Examples of a panoramic camera can include asingle camera, a polycamera, a camera rosette, a rotating camera, etc.The captured images may be used for online navigation and viewing toolssuch as Google Inc.'s STREET VIEW tool, for example. Vehicles that usepanoramic camera systems in this manner may also include other systemsand devices for related data collection. For example, a data collectionvehicle may include a Global Positioning System (GPS) and/or an InertialMeasurement Unit (IMU) sensor in addition to the camera system. It mayalso record the amount of rotation of the vehicle's wheels. Thesesystems include sensors that can collect data, which can help estimatethe location of the vehicle. Given the precise location of the vehicle,the captured images can be associated with and shown at those locations.

There is nontrivial variation in the way a panoramic camera system andGPS and IMU sensors are placed on, or within, a data collection vehicle.For example, there is little consistency in the placement of a camerarack on top of the vehicle roof. In addition, there is variation in howand where the GPS and IMU sensors are placed within the vehicle.Furthermore, cameras and camera racks are often replaced, or theirconfiguration and/or positioning may be changed by human operators. Inmany applications, in order to correctly render a panoramic view, oneneeds to know how the ground plane and world coordinates relate to theimage panorama that was captured by the panoramic camera. If thisinformation is not known or inaccurate, objects (e.g., buildings) andtheir surroundings may appear incorrectly, e.g., tilted to one side.Furthermore, directional arrows that may be used in a viewing tool maypoint in a wrong direction. Thus, knowing the camera orientationrelative to GPS and/or IMU sensors in a data collection vehicle can beimportant.

BRIEF SUMMARY

Embodiments of the invention relate to estimation of camera orientationrelative to a vehicle coordinate frame. In one embodiment, a method forestimating orientation of a panoramic camera mounted on a vehicle mayinclude determining an initial pose estimate of the vehicle based onglobal positioning system data, inertial measurement unit data, andwheel odometry data of the vehicle. The method may also includeobtaining images from one or more runs of image data captured by thecamera, the images each having an orientation. The method may furtherinclude processing image data from the images to obtain one or moretracks, where each track includes a sequence of matched feature pointsstemming from a same three-dimensional location. The method may alsoinclude determining, from the initial pose estimate and tracks, acorrection parameter to correct the orientations of the images capturedby the camera.

In another embodiment, a system for estimating orientation of apanoramic camera mounted on a vehicle is provided. The system mayinclude a pose estimate module that generates an initial pose estimateof the vehicle based on global positioning system data, inertialmeasurement unit data, and wheel odometry data of the vehicle. Thesystem may also include an image processing module that processes imagedata from one or more runs of image data captured by the camera toobtain one or more tracks, where each track includes a sequence ofmatched feature points stemming from a same three-dimensional location.The system may further include an optimizer module, in communicationwith the pose estimate module and the image processing module, thatdetermines, from the initial pose estimate and tracks, a correctionparameter to correct the orientations of the images. In an embodiment,the pose estimate module may be in communication with one or morevehicle databases having vehicle information such as global positioningsystem data, inertial measurement unit data, and wheel odometry data ofthe vehicle. In an embodiment, the image processing module may be incommunication with one or more image databases having images andcorresponding image data from the one or more runs of image datacaptured by the panoramic camera. In an alternative embodiment, thesystem can include the vehicle databases and/or the image databases.

In one embodiment, a computer program product includes a computerreadable storage medium having control logic stored therein for causinga computer to estimate orientation of a panoramic camera mounted on avehicle. The control logic may include a first computer readable programcode that enables the computer to determine an initial pose estimate ofthe vehicle, the initial pose estimate based on global positioningsystem data, inertial measurement unit data, and wheel odometry data ofthe vehicle. The control logic may also include a second computerreadable program code that enables the computer to obtain images fromone or more runs of image data captured by the camera, the images eachhaving an orientation. The control logic may further include a thirdcomputer readable program code that enables the computer to processimage data from the images to obtain one or more tracks, where eachtrack includes a sequence of matched feature points stemming from a samethree-dimensional location. The control logic may also include a fourthcomputer readable program code that enables the computer to determine,from the initial pose estimate and tracks, a correction parameter tocorrect the orientations of the images captured by the camera.

Further embodiments, features, and advantages, as well as the structureand operation of the various embodiments, are described in detail belowwith reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

The accompanying drawings, which are incorporated herein and form partof the specification, illustrate the present invention and, togetherwith the description, further serve to explain the principles of theinvention and to enable a person skilled in the relevant art(s) to makeand use the invention.

FIGS. 1-3 are exemplary diagrams depicting a vehicle with a panoramiccamera mounted thereon.

FIG. 4 shows exemplary ways of representing roll.

FIG. 5 is a block diagram depicting inputs and output of a system forestimating a camera orientation, according to an embodiment of thepresent invention.

FIG. 6 is a diagram depicting an example of feature matching.

FIG. 7 is a diagram depicting an example of pose, rotation, andthree-dimensional location.

FIG. 8 shows a nonlinear equation for deriving a camera orientationcorrection parameter, according to an embodiment of the presentinvention.

FIGS. 9 and 10 depict ways of improving the camera orientationcorrection parameter that may be determined from the equation shown inFIG. 8, according to embodiments of the present invention.

FIGS. 11A and 11B are block diagrams depicting systems for estimating acamera orientation, according to embodiments of the present invention.

FIG. 12 is a flowchart depicting a method for estimating orientation ofa camera mounted on a vehicle, according to an embodiment of the presentinvention.

FIG. 13 is a flowchart depicting an optional further step of the methodshown in FIG. 12, according to an embodiment of the present invention.

FIG. 14 is a flowchart depicting optional further steps of the methodshown in FIG. 12, according to an embodiment of the present invention.

FIG. 15 is a flowchart depicting an example of step 1414 of the methodshown in FIG. 14, according to an embodiment of the present invention.

FIG. 16 is a flowchart depicting optional further steps of the methodshown in FIG. 12, according to an embodiment of the present invention.

FIG. 17 is a flowchart depicting an example of step 1620 of the methodshown in FIG. 16, according to an embodiment of the present invention.

FIG. 18 is an exemplary block diagram of a computer system that can beused to implement embodiments of the present invention.

The features and advantages of the present invention will become moreapparent from the detailed description set forth below when taken inconjunction with the drawings, in which like reference charactersidentify corresponding elements throughout. In the drawings, likereference numbers generally indicate identical, functionally similar,and/or structurally similar elements. The drawing in which an elementfirst appears is indicated by the leftmost digit(s) in the correspondingreference number.

DETAILED DESCRIPTION OF EMBODIMENTS

While the present invention is described herein with reference toillustrative embodiments for particular applications, it should beunderstood that the invention is not limited thereto. Those skilled inthe art with access to the teachings provided herein will recognizeadditional modifications, applications, and embodiments within the scopethereof and additional fields in which the invention would be ofsignificant utility.

It is noted that references in the specification to “one embodiment,”“an embodiment,” “an example embodiment,” etc., indicate that theembodiment described may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same embodiment. Further, when aparticular feature, structure, or characteristic is described inconnection with an embodiment, it would be within the knowledge of oneskilled in the art to incorporate such a feature, structure, orcharacteristic in connection with other embodiments whether or notexplicitly described.

Terminology

The following provides definitions for certain terms as used in thisdocument:

Panoramic Camera—A panoramic camera is a camera, or system of cameras,that is configured or arranged to capture an image or view of an area inone or many directions. Examples of a panoramic camera may include asingle camera, a polycamera (a tightly-packed cluster of camerasproviding a large field of view), a camera rosette (outward-facingequally-spaced cameras forming a circle that provide an all-aroundview), a rotating (or rotating line) camera (a camera that is rotated tocapture images in multiple directions), etc.

GPS—Global Positioning System—The Global Positioning System (GPS) is anavigational system using satellite signals to determine the location ofa radio receiver on or above the earth's surface.

IMU—Inertial Measurement Unit—An Inertial Measurement Unit is aposition-tracking sensor that senses motion in terms of type, rate, anddirection using a combination of accelerometers and gyroscopes.

World Coordinates—World coordinates are from the world coordinatesystem, which is a coordinate system that is fixed with respect to theEarth.

Pose—A pose may be defined as a three-dimensional position (e.g., in thex, y, z coordinate system) with an orientation (or rotation) that isusually referred to using rotation coordinates (e.g., roll (φ), pitch(θ), and yaw (Ψ)). Therefore, a pose may be expressed in at least sixdimensions: x, y, z, φ, θ, and Ψ. The pose of the vehicle may be definedas a position and orientation of the vehicle relative to the world.

Feature Matching—Feature matching provides correspondence betweenfeature points and images. Detected features from different cameraimages are matched using their appearance to find corresponding sets offeatures. Each set of matching features is assumed to be produced by thesame entity, which has a certain three-dimensional position in theworld. Matched feature points may be grouped into one or more tracks,each track including a sequence of matched feature points stemming froma single three-dimensional location. Feature detection and matching maybe used for image alignment (e.g., stitching), three-dimensionalreconstruction, motion tracking, etc.

Overview

Embodiments of the present invention are related to panoramicphotography via a panoramic camera that is mounted on a vehicle. Theembodiments are directed to optimizing orientation of images obtainedvia such a camera. Because the alignment of the camera may not be ideal,and the physical orientation, position, and/or location of the cameramay not align with those of other related data collection sensors, suchas a Global Positioning System (GPS) and/or an Inertial Measurement Unit(IMU) sensor, images obtained via the camera may be improved by applyingthe embodiments described herein. For example, embodiments may includethe ability to automatically estimate the orientation of a cameramounted on a data collection vehicle relative to the data collectionsensors.

The following description is broken down into a discussion ofenvironment, orientation estimation, further optimization of orientationcorrection, system architecture, and methods of orientation estimationand optimization.

Environment

FIGS. 1-3 are exemplary diagrams depicting a vehicle 102 with apanoramic camera 104 mounted thereon. Panoramic camera 104 may be anytype of panoramic camera as described earlier, e.g., a single camera, apolycamera, a camera rosette, etc. Panoramic camera 104 may include anynumber of cameras 106. In one example, panoramic camera 104 includesfour to eight cameras. However, this is illustrative and not to belimiting, as fewer or more cameras may be used. Vehicle 102 may be usedto collect image data and other data for navigation and viewing toolssuch as Google Inc.'s STREET VIEW tool.

In addition to panoramic camera 104, other sensors (not shown) may beused for data collection, such as, for example, a Global PositioningSystem (GPS) sensor and/or an Inertial Measurement Unit (IMU) sensor.These other sensors, as well as other related equipment, may be locatedin the trunk 108 of vehicle 102, or anywhere else within, on, or coupledto vehicle 102. In example 100 of FIG. 1, image 110 represents anexample image obtained from panoramic camera 104. Each divided portion(denoted by dotted lines), may show an image captured by each of fourcameras of panoramic camera 104, for example. In example 200 shown inFIG. 2, panoramic camera 104 is mounted in a skewed manner, and objectsshown in image 210 obtained from camera 104 are tilted and/or displacedupward or downward relative to ground 101. As can be seen in FIG. 2,each image has a different displacement, depending on which camera ofpanoramic camera 104 captured each image. This is one example ofincorrect image orientation due to a physically skewed camera. Anotherexample may be that camera 104 is mounted on vehicle 102 in an evenmanner, but vehicle 102 is tilted. However, even if camera 104 was notphysically skewed, image 210 may still contain skewed contents due toother conditions, such as inconsistent or unknown placement of relateddata collection sensors (e.g., a GPS sensor or an IMU sensor). Sincesensor data is associated with captured images, this inconsistent orunknown placement of sensors can lead to skew in processing of theimages to form a skewed panoramic image. Furthermore, a combination ofthese conditions may exist. As stated earlier, knowing the cameraorientation relative to GPS and/or IMU sensors in a data collectionvehicle can be important, as will now be discussed.

Generally, any three-dimensional object can be considered as having asix-dimensional pose. A pose may be defined as a three-dimensionalposition (e.g., in the x, y, z coordinate system) with an orientation(or rotation) that is usually referred to using rotation coordinates(e.g., roll (φ), pitch (θ), and yaw (Ψ)), such as coordinate set 413shown in FIG. 4, although rotation may be represented in other ways(such as quaternion 415 shown in FIG. 4). Thus, a pose may be expressedin at least six dimensions: x, y, z, φ, θ, and Ψ. Referring to FIG. 3,the pose of a moving vehicle may be represented by, for example, aseries of poses 312 of data collection vehicle 102. Each pose of theseries of poses 312 is shown by a schematic representation including acircle with three arrows. This representation is meant to representposition (x, y, z) and orientation (φ, θ, Ψ).

The pose of a moving vehicle may be defined in the above-describedmanner, with the coordinates constantly changing as the vehicle moves ina three-dimensional space and along uneven terrain (e.g., hilly and/orwindy roads). Sensors placed in or on vehicle 102, such as GPS and/orIMU sensors, may assist in determining a pose of vehicle 102. Generally,GPS sensors use satellite data to determine location, speed, direction,and time. IMU sensors generally include a combination of accelerometersand gyroscopes, and may determine position by sensing acceleration androtational attributes. When used as part of a navigational viewcapturing system, these sensors may provide information to determine howthe ground plane and the world coordinates relate to an image, or imagepanorama, that is captured by one or more cameras of panoramic camera104. This information is used to correctly render a panoramic view, forexample. If this information is not known or inaccurate, objects (e.g.,buildings, trees, etc.) and their surroundings may appear incorrectly orskewed, e.g., tilted to one side and/or displaced upward or downwardrelative to ground 101. Correction may then require orientationadjustment of the raw and/or rendered images. With a potentialadditional problem of mounted camera 104 not being aligned in a straightmanner, this adjustment becomes even more important.

The above-discussed problems may be corrected by the embodimentsdiscussed in the following description.

Orientation Estimation and Optimization

Embodiments as described below rely on the assumption that relativelyaccurate estimates of vehicle pose, and images for all cameras of apanoramic camera 104 mounted on the vehicle, are available for anuninterrupted data collection interval or run. With this data, accuraterotational alignment between the panoramic camera 104 and GPS/IMUsensors that were used to obtain the vehicle pose estimates may bedetermined.

An embodiment may include an orientation estimating system 517, as shownin FIG. 5. Orientation estimating system 517 may include a computingdevice 541. Computing device 541 can be any type of computing devicehaving one or more processors. For example, computing device 541 can bea workstation, mobile device, computer, cluster of computers, set-topbox, or other device having at least one processor. Such a computingdevice may include software, firmware, hardware, or a combinationthereof. Software may include one or more applications and an operatingsystem. Hardware can include, but is not limited to, a processor and amemory. Hardware may also include a graphical user interface display.

According to an embodiment, the orientation estimating system 517 mayreceive vehicle pose data 519, that may include GPS, IMU, and thevehicle's wheel odometry data, and image data 521, that may includeimage data obtained by a panoramic camera 104 mounted on the vehicle. Inan embodiment, the orientation estimating system 517 may determinevehicle pose estimates based on vehicle pose data 519, or alternatively,vehicle pose estimates may be provided to orientation estimating system517 as part of the vehicle pose data 519. In an embodiment, theorientation estimating system 517 may determine image track data basedon image data 521, using feature matching for example, or alternatively,image track data may be provided to orientation estimating system 517 aspart of the image data 521. The orientation estimating system 517 maythen determine one or more correction parameters 523 that may be appliedto image data 521, for example, to provide correctly oriented views, asdiscussed in the following paragraphs.

According to an embodiment, the determination of one or more correctionparameters 523 may be accomplished by applying an orientation estimationalgorithm to the vehicle pose estimates and image track data obtainedvia feature matching. The orientation estimation algorithm may be usedto improve vehicle pose estimates, estimates of the three-dimensionallocations of the entities used for the feature matching, and estimatesof the camera orientation relative to the GPS/IMU sensors that were usedto obtain the vehicle's pose estimates.

The feature matching in the captured images may be accomplished usingknown feature matching techniques. For various embodiments, a set offeatures (e.g., scale-invariant salient points on an image, where a lotof texture is present) are detected in images captured by a panoramiccamera. The detected features from different camera images (e.g.,captured at different times) are matched using their appearance to findcorresponding sets of features. Each set of matching features is assumedto be produced by the same entity, which has a certain three-dimensionalposition in the world. Matched feature points may be grouped into one ormore tracks, each track including a sequence of matched feature pointsstemming from a single three-dimensional location. An example of aportion of the feature matching process is shown in FIG. 6. In FIG. 6, afeature point 627 is detected in different images 629A, 629B, and 629Cas detected feature point 631.

In an embodiment, the key parameters involved with the orientationestimation algorithm are vehicle pose (P), three-dimensional locationsof the entities used for feature matching (X), and camera orientationrelative to the GPS/IMU sensors (R). These parameters are demonstratedin FIG. 7. For a vehicle (not shown) moving along roadways 735, thevehicle's pose (P) at different locations is represented by points 737.The three-dimensional locations (X) of entities “viewed” from thevehicle are represented by points 739. The orientation R of a camera(not shown) mounted on the vehicle relative to GPS/IMU sensors locatedin, on, or coupled to the vehicle is the same for all poses 737.

In an embodiment, the orientation estimation algorithm is based onEquation 800, shown in FIG. 8. Equation 800 is a nonlinear functiondefined by:

F(P,X,R)=Σ_(t)Σ_(i)ρ((T _(Pi,R)(X _(t))−I _(Xt)))²+λΣ_(i)(P _(i) −P_(ESTi))²

where

-   -   P=P₁, P₂, . . . , P_(N) and represents a set of vehicle poses;    -   P_(i) represents a pose of the vehicle at time i;    -   X=X₁, X₂, . . . , X_(M) and represents three-dimensional        locations of track points in a scene;    -   X_(t) represents a three-dimensional location of a track t in        the scene;    -   R represents the rotation of the camera;    -   ρ denotes a robustifier function (e.g., a Cauchy robustifier);    -   T represents projection;    -   I_(Xt) represents a fixed location in a given image, where a        feature corresponding to track point X_(t) was detected;    -   λ represents a weight used to trade off strength of a first and        a second term in F; and    -   P_(ESTi) represents an initial or a previous pose estimate of        the vehicle.

In an embodiment, Equation 800 may be used to determine estimates ofvehicle pose P, entity locations X, and camera orientation R. Multipleiterations of Equation 800 may provide improved estimates of P, X, andR. Estimated camera orientation R may be applied to image data of imagescaptured by the camera 104 to correct their orientation so that they maybe more accurately viewed. For example, the camera orientation R may beapplied to the image data at a point when the image is stitched.

In Equation 800, T_(Pi, R)(X_(t))−I_(Xt) represents reprojection errorand P_(i)−P_(ESTi) represents pose error. Reprojection error is ageometric error that corresponds to the image distance between aprojected point and a measured point. It is used to quantify how closelyan estimate of a three-dimensional point recreates the point's trueprojection. In FIG. 6, reprojection error is shown in an exaggeratedway, for clarity, as distance 633. According to an embodiment, usingEquation 800, parameters P, X, and R are to be determined such that theyminimize reprojection error. In other words, the projections of thethree-dimensional entities into an image should overlap with the imagefeatures that were detected with feature matching. At the same time, thepose should remain as similar to the original pose as possible. Thus,Equation 800 allows one to determine the relative rotational alignmentthat best determines camera orientation and that minimizes reprojectionerror with the original set of pose estimates.

The minimization of the objective in Equation 800 can be performed withany standard non-linear optimization technique, such as but notconstrained to Levenberg-Marquardt, Conjugate Gradient, or gradientdescent methods.

Assuming an accurate set of initial vehicle pose estimates determinedusing GPS, IMU, and wheel odometry data is used, the orientationestimation algorithm described above provides the rotation between theinitial vehicle pose estimates (dependent on the GPS/IMU coordinatesystems) and the poses that minimize reprojection error (dependent onthe coordinate system of the camera).

In the embodiments described above, a camera rotation correctionparameter is determined for a particular portion, or subsequence, of arun. The quality of the result, however, is dependent on the quality ofthe initial vehicle pose estimate that one may determine from wheelodometry data and the data from GPS and IMU sensors. If the initialvehicle pose estimate is inaccurate due to errors in the GPS, IMU,and/or wheel odometry inputs, the camera rotation correction parametermay also be inaccurate. The following section discusses ways to make thecamera rotation correction parameter more robust in accordance withvarious embodiments.

Further Optimization of Orientation Correction

In the above-described embodiments, a camera rotation correctionparameter is determined for a single subsequence of a given, orselected, run. According to one embodiment, the rotation correctionparameter may be made more robust by analyzing multiple subsequences ofa selected run. For example, in one embodiment, multiple rotationcorrection parameters may be determined, as described above, for amultitude of subsequences of a selected run, and statistical analysis,possibly with outlier removal, may be performed on the determinedcorrection parameters to determine an optimized correction parameter.For example, in an embodiment, a median of the determined correctionparameters may be determined and used as an optimized correctionparameter.

Optionally, from a multitude of determined correction parameters,correction parameters that appear to be very different from the rest maybe ignored or removed from the analysis. As an example, in oneembodiment, correction parameters may be ignored for subsequences of theselected run in which the acceleration of the vehicle is above apredetermined value or outside of a given range. As a further example,in one embodiment, correction parameters may be ignored for subsequencesof the selected run in which the vehicle is moving outside of apredetermined velocity range. In yet another example, in one embodiment,a cost function may be used, possibly within a dynamic programmingalgorithm that chooses subsequences of the selected run that abide withone or more given rules (e.g., having a vehicle acceleration that iswithin a given range, having a vehicle velocity that is within a givenrange, etc.).

FIG. 9 demonstrates an example of improving the camera orientationcorrection parameter as described above, in accordance with variousembodiments. In FIG. 9, a run 950, representing a run of a datacollection vehicle for example, is shown as consisting of many distinctsubsequences 952, each with their own estimated pose P and rotationcorrection parameter R. In the example shown, subsequences 952 haveoverlap. However, it is not necessary for subsequences 952 to overlap.Instead, subsequences 952 may be sequential, for example. In the exampleshown, a median 954 of the estimates of rotation correction parameter Ris determined, which may be used as an optimized rotation correctionparameter R. As discussed above, rotation correction parameter R may bemade even more robust by omitting outliers in its calculation. Forexample, it may be desirable to omit estimates R_(j) and R_(m) from thedetermination of median 954 because they are associated withsubsequences where the vehicle's velocity was unacceptable and where thevehicle's acceleration was unacceptable, respectively. The example shownin FIG. 9 is just one example. It will be appreciated by those skilledin the art that other similar ways of optimizing rotation correctionparameter R may be realized within the scope of the present invention.

In an embodiment, the above optimization may be accomplished usinginformation gathered over multiple runs. In one embodiment, for example,multiple rotation correction parameters may be determined, as describedabove, for a multitude of subsequences of multiple runs. Statisticalanalysis, possibly with outlier removal, may be performed on thedetermined correction parameters to determine an optimized correctionparameter, as previously described above. FIG. 10 demonstrates anexample showing multiple subsequences of multiple runs, in accordancewith an embodiment. In FIG. 10, a run 1050 is shown consisting of manydistinct subsequences 1052, and a run 1054 is shown consisting of manydistinct subsequences 1056, each with their own estimated pose P androtation correction parameter R (e.g., P_(A0), R_(A0)-P_(AZ), R_(AZ) forrun 1050, and P_(B0), R_(B0)-P_(BZ), R_(BZ) for run 1054). Additionalruns could also exist.

In an embodiment, an optimized rotation correction parameter R may bedetermined by, for each run of a multitude of runs, determining a firstmedian of the determined rotation correction parameters for each of theclosest Z runs backward in time, and determining a second median of thedetermined rotation correction parameters for each of the closest Z runsforward in time. Either the first or second median may be chosen as theoptimized rotation correction parameter based on which of the first orsecond median is closest to the rotation correction parameter determinedfor that run.

The analysis and determination of optimized rotation correctionparameters can be done by orientation estimating system 517, describedpreviously.

System Architecture

FIGS. 11A and 11B are block diagrams depicting systems 1117A and 1117B,respectively, for estimating a camera orientation, according toembodiments. Systems 1117A and 1117B may each be substituted as system517 shown in FIG. 5.

Systems 1117A and 1117B may include a pose estimate module 1174, animage processing module 1176, and an optimizer module 1178. In anembodiment, each of the pose estimate module 1174, image processingmodule 1176, and optimizer module 1178 may include one or moreprocessors of one or more computing devices, such as computing device541 shown in FIG. 5 and discussed above. In one embodiment, poseestimate module 1174, image processing module 1176, and optimizer module1178 may each be implemented as one or more processors of a singlecomputing device. Pose estimate module 1174, image processing module1176, and optimizer module 1178 may be implemented using software,firmware, hardware, or a combination thereof.

With reference to FIG. 11A, according to an embodiment, pose estimatemodule 1174 may receive or obtain vehicle-related data 519 for aparticular vehicle, such as wheel odometry data, GPS-related data, andIMU-related data. Pose estimate module 1174 may use vehicle-related data519 to determine pose estimates 1180 for the vehicle.

According to an embodiment, image processing module 1176 may receive orobtain images and related image data 521 obtained from a panoramiccamera 104 mounted on the vehicle. Image processing module 1176 mayconduct feature matching based on images and related image data 521 todetermine image track data 1182 related to the three-dimensionallocations of the entities used for feature matching.

According to an embodiment, optimizer module 1178 may determine, forexample using Equation 800 defined above, estimates of vehicle pose (P),three-dimensional location of the entities used for feature matching(X), and a rotation correction parameter (R) based on pose estimates1180 and image track data 1182. The P, X, and R estimates 1184 may beoutput for use by another system (not shown) or stored in a data storeor database (not shown).

System 1117B is similar to system 1117A, except that system 1117Bincludes one or more databases for the vehicle data and one or moredatabases for the image information as part of the camera orientationestimating system. In an embodiment, vehicle database 1170 may includevehicle-related data 1162 for a particular vehicle, such as wheelodometry data, GPS-related data, and IMU-related data. Pose estimatemodule 1174 may use vehicle-related data 1162 to determine poseestimates 1180 for the vehicle.

In an embodiment, image database 1172 may include images and relatedimage data 1164 obtained from a camera 104 mounted on the vehicle. Imageprocessing module 1176 may conduct feature matching based on images andrelated image data 1164 that it receives from image database 1172 todetermine image track data 1182 related to the three-dimensionallocations of the entities used for feature matching.

As previously described, optimizer module 1178 of system 1117B maydetermine, for example using Equation 800 defined above, estimates ofvehicle pose (P), three-dimensional location of the entities used forfeature matching (X), and a rotation correction parameter (R) based onpose estimates 1180 and image track data 1182. In an embodiment, the P,X, and R estimates 1184 may be stored in a data store, such as imagedatabase 1172, for example, or another storage location (not shown). Inanother embodiment, the P, X, and R estimates 1184 may be output asshown in FIG. 1117A.

Rotation correction parameter R may be applied to the images stored inimage database 1172 for accurate viewing. In an embodiment, theapplication of rotation correction parameter R to a particular image maybe done, for example, via a computer system, or processing module (suchas an orientation correction module 1186 shown in FIG. 11A) duringstitching of the image. In an embodiment, orientation correction module1186 may be a part of system 1117A or 1117B. In another embodiment,orientation correction module 1186 may be in communication with, butseparate from, system 1117A or 1117B. In one embodiment, orientationcorrection module 1186 can obtain correction parameter information fromoptimizer module 1178. In another embodiment, orientation correctionmodule 1186 can obtain correction parameter information that is storedin image database 1172 (shown in FIG. 11B).

Methods

FIGS. 12-17 are flowcharts depicting methods for estimating andoptimizing orientation of a camera 104 mounted on a vehicle, accordingto embodiments of the present invention. According to an embodiment, instep 1202 of method 1200 (FIG. 12), an initial pose estimate of avehicle is determined based on global positioning system data, inertialmeasurement unit data, and wheel odometry data of the vehicle. In step1204, images are obtained from one or more runs of image data capturedby the camera 104. In step 1206, image data from the images is processedto obtain one or more tracks. Each track includes a sequence of matchedfeature points stemming from a single three-dimensional location. Instep 1208, a correction parameter is determined from the initial poseestimate and tracks. Method 1200 then ends. In this way, a correctionparameter for correcting skew of one or more images obtained by camera104 is automatically determined. One or more of the steps of method 1200may be performed by camera orientation estimating system 517 (or1117A/B).

FIG. 13 is a flowchart depicting an optional further step of method1200, according to an embodiment. In step 1310, which may stem from step1208, orientations of the images are corrected by applying thecorrection parameter to the image data. In this way, a view of the imagewill appear correctly oriented even if the original image was skewed. Inan embodiment, step 1310 may be performed by camera orientationestimating system 517 (or 1117A/B) or another processor or computingsystem, for example.

FIG. 14 is a flowchart depicting optional further steps of method 1200,according to an embodiment. In step 1412, which may stem from step 1208,the correction parameter for each of a multitude of distinctsubsequences of a selected run is determined. In an embodiment, thesubsequences may overlap. In another embodiment, the subsequences may beserial. In step 1414, statistical analysis, and possibly outlierremoval, is performed on the determined correction parameters to obtainan optimized correction parameter, as discussed above with reference toFIG. 9.

In an embodiment, an example of step 1414 is shown in step 1516 of theflowchart in FIG. 15. In step 1516, a median value of the determinedcorrection parameters is determined. This median value may be used asthe optimized correction parameter. When applied to the image data, theoptimized correction parameter will provide improved image orientationcorrection over a correction parameter that does not take smallersubsequences of a run into account.

FIG. 16 is a flowchart depicting optional further steps of method 1200,according to an embodiment. In step 1618, which may stem from step 1208,the correction parameter for each of a multitude of distinctsubsequences of a multitude of runs is determined. In step 1620,statistical analysis, and possibly outlier removal, is performed on thedetermined correction parameters across the multitude of runs to obtainan optimized correction parameter, as discussed above with reference toFIG. 10.

In an embodiment, an example of step 1620 is shown in the flowchart inFIG. 17. In step 1722, for each run of the multitude of runs, a firstmedian of the correction parameters is determined for each of theclosest Z runs backward in time and a second median of the correctionparameters is determined for each of the closest Z runs forward in time.In step 1724, either the first or second median may be chosen as theoptimized correction parameter for a selected run based on which of thefirst or second median is closest to the correction parameter determinedfor that run. For example, if there were 100 runs, for a selected run, afirst median may be determined for each of the closest 10 runs backwardin time and a second median may be determined for each of the closest 10runs forward in time. The median value that is closest to the determinedcorrection parameter for that run may be chosen as the optimizedcorrection parameter.

Exemplary Computer System

The various embodiments described herein may be implemented usinghardware, software or a combination thereof and may be implemented in acomputer system or other processing system. In an embodiment, theinvention is directed toward a computer program product executing on acomputer system capable of carrying out the functionality describedherein. An example of a computer system 1800 is shown in FIG. 18. Theorientation estimating systems 517 (of FIGS. 5) and 1117A/B (of FIGS.11A and 11B) described above could be implemented in a computer systemincluding, but not limited to, computer system 1800. The computer system1800 includes one or more processors, such as processor 1804. Processor1804 may be a general purpose processor (such as, a CPU) or a specialpurpose processor (such as, a GPU). Processor 1804 is connected to acommunication infrastructure 1806 (e.g., a communications bus,cross-over bar, or network). Various software embodiments are describedin terms of this example computer system. After reading thisdescription, it will become apparent to a person skilled in the relevantart how to implement the invention using other computer systems and/orcomputer architectures.

Computer system 1800 (optionally) includes a display interface 1802(which can include input/output devices such as keyboards, mice, etc.)that forwards graphics, text, and other data from communicationinfrastructure 1806 (or from a frame buffer not shown) for display ondisplay unit 1830.

Computer system 1800 also includes a main memory 1808, preferably randomaccess memory (RAM), and may also include a secondary memory 1810. Thesecondary memory 1810 may include, for example, a hard disk drive 1812and/or a removable storage drive 1814, representing a floppy disk drive,a magnetic tape drive, an optical disk drive, etc. The removable storagedrive 1814 reads from and/or writes to a removable storage unit 1818 ina well-known manner. Removable storage unit 1818, represents a floppydisk, magnetic tape, optical disk, memory card, etc. which is read byand written to by removable storage drive 1814. As will be appreciated,the removable storage unit 1818 includes a computer readable storagemedium having stored therein computer software and/or data.

In alternative embodiments, secondary memory 1810 may include othersimilar means for allowing computer programs or other instructions to beloaded into computer system 1800. Such means may include, for example, aremovable storage unit 1822 and an interface 1820. Examples of such mayinclude a program cartridge and cartridge interface (such as that foundin video game devices), a removable memory chip (such as an EPROM, orPROM) and associated socket, and other removable storage units 1822 andinterfaces 1820 which allow software and data to be transferred from theremovable storage unit 1822 to computer system 1800.

Computer system 1800 may also include a communication interface 1824.Communication interface 1824 enables computer 1800 to communicate withexternal and/or remote devices. For example, communication interface1824 allows software and data to be transferred between computer system1800 and external devices. Communication interface 1824 also allowscomputer 1800 to communicate over communication networks, such as LANs,WANs, the Internet, etc. Communication interface 1824 may interface withremote sites or networks via wired or wireless connections. Examples ofcommunications interface 1824 may include a modem, a network interface(such as an Ethernet card), a communications port, a PCMCIA slot andcard, etc. Computer 1800 receives data and/or computer program productsvia communication network 1824. Software and data transferred viacommunications interface 1824 are in the form of signals 1828 which maybe electronic, electromagnetic, optical or other signals capable ofbeing received by communications interface 1824. These signals 1828 areprovided to communications interface 1824 via a communications path(i.e., channel) 1826. This channel 1826 carries signals 1828 and may beimplemented using wire or cable, fiber optics, a phone line, a cellularphone link, an RF link and other wired or wireless communicationschannels.

In this document, the terms “computer program medium” and “computerusable medium” and “computer readable medium” are used to generallyrefer to media such as removable storage drive 1814, and a hard diskinstalled in hard disk drive 1812. These computer program products aremeans for providing software to computer system 1800.

Computer programs (also called computer control logic) are stored inmain memory 1808 and/or secondary memory 1810. Computer programs mayalso be received via communications interface 1824. Such computerprograms, when executed, enable the computer system 1800 to perform thefeatures of the present invention as discussed herein. In particular,the computer programs, when executed, enable the processor 1804 toperform the features of the present invention. Accordingly, suchcomputer programs represent controllers of the computer system 1800.

In an embodiment implemented using software, the software may be storedin a computer program product and loaded into computer system 1800 usingremovable storage drive 1814, hard disk drive 1812 or communicationsinterface 1824. The control logic (software), when executed by theprocessor 1804, causes the processor 1804 to perform the functions ofthe invention as described herein.

The invention can work with software, hardware, and operating systemimplementations other than those described herein. Any software,hardware, and operating system implementations suitable for performingthe functions described herein can be used.

Conclusion

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation and without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

The Summary and Abstract sections may set forth one or more but not allexemplary embodiments of the present invention as contemplated by theinventor(s), and thus, are not intended to limit the present inventionand the appended claims in any way.

Further, the purpose of the foregoing Abstract is to enable the U.S.Patent and Trademark Office, the public generally, and especially thescientists, engineers and practitioners in the art who are not familiarwith patent or legal terms or phraseology, to determine quickly from acursory inspection the nature and essence of the technical disclosure ofthe application. The Abstract is not intended to be limiting as to thescope of the present invention in any way.

1. A method for estimating orientation of cameras mounted on vehicles,comprising: determining, by one or more processors, an initial poseestimate of a vehicle having a camera mounted thereon, the initial poseestimate being a position and orientation of the vehicle relative to theworld; processing, by the one or more processors, image data from imagescaptured by the camera to obtain one or more tracks, each trackincluding a sequence of matched feature points stemming from a singlethree-dimensional location; and determining, by the one or moreprocessors, from the initial pose estimate and the tracks, a correctionparameter to correct orientations of the images captured by the camera,wherein determining the correction parameter includes evaluating a setof vehicle poses, three-dimensional locations from the one or moretracks, and an orientation of the camera relative to one or more sensorsof the vehicle.
 2. The method of claim 1, wherein determining thecorrection parameter is done such that a reprojection error isminimized.
 3. The method of claim 1, wherein the correction parameter isa camera rotation correction parameter that is determined for aparticular portion of an uninterrupted data collection interval.
 4. Themethod of claim 1, wherein the set of vehicle poses is obtained fromlocation data, inertial measurement data, and wheel odometry data of thevehicle.
 5. The method of claim 1, further comprising correctingorientations of the images by applying the correction parameter to theimage data.
 6. The method of claim 5, wherein the determining thecorrection parameter includes determining: an optimized pose of thevehicle; a location of points of the tracks in three dimensions; and acamera to vehicle pose rotation.
 7. The method of claim 1, wherein: thedetermining includes determining the correction parameter for each of amultitude of subsequences of a selected run; and performing statisticalanalysis on the determined correction parameters for differentsubsequences to determine an optimized correction parameter.
 8. Themethod of claim 7, wherein performing the statistical analysis includesdetermining a median value of the determined correction parameters touse as the optimized correction parameter.
 9. The method of claim 8,wherein determining the median value omits determined correctionparameters for subsequences of the selected run in which acceleration ofthe vehicle is above a predetermined value or outside of a predeterminedrange.
 10. The method of claim 8, wherein determining the median valueomits determined correction parameters for subsequences of the selectedrun in which the vehicle is moving outside of a predetermined velocityrange.
 11. The method of claim 7, wherein determining the correctionparameter for each of a multitude of subsequences of a selected runincludes performing a cost function to chooses subsequences of theselected run that conform to a predetermined rule.
 12. The method ofclaim 1, wherein: the determining includes determining the correctionparameter for each of a multitude of subsequences of a multitude ofruns; and performing statistical analysis on the determined correctionparameters across the multitude of runs to determine an optimizedcorrection parameter.
 13. The method of claim 12, wherein the performingstatistical analysis includes: for each run of the multitude of runs,determining a first median of the correction parameters for each of aclosest set of runs backward in time and determining a second median ofthe correction parameters for each of a closest set of runs forward intime; and selecting either the first median or the second median as theoptimized correction parameter based on which of the first or secondmedian is closest to the correction parameter determined for that run.14. A system, comprising: one or more processors configured to:determine an initial pose estimate of a vehicle having a camera mountedthereon, the initial pose estimate being a position and orientation ofthe vehicle relative to the world; process image data from imagescaptured by the camera to obtain one or more tracks, each trackincluding a sequence of matched feature points stemming from a singlethree-dimensional location; and determine, from the initial poseestimate and the tracks, a correction parameter to correct orientationsof the images captured by the camera, wherein determining the correctionparameter includes evaluating a set of vehicle poses, three-dimensionallocations from the one or more tracks, and an orientation of the camerarelative to one or more sensors of the vehicle.
 15. The system of claim15, wherein the one or more processors are further configured to correctorientations of the images by applying the correction parameter to theimage data.
 16. The system of claim 15, wherein the one or moreprocessors are configured to determine the correction parameter bydetermining: an optimized pose of the vehicle; a location of points ofthe tracks in three dimensions; and a camera to vehicle pose rotation.17. The system of claim 15, wherein: the one or more processors areconfigured to determine the correction parameter for each of a multitudeof subsequences of a selected run; and the one or more processors arefurther configured to perform statistical analysis on the determinedcorrection parameters for different subsequences to determine anoptimized correction parameter.
 18. The system of claim 15, wherein: theone or more processors are configured to determine the correctionparameter for each of a multitude of subsequences of a multitude ofruns; and the one or more processors are further configured to performstatistical analysis on the determined correction parameters across themultitude of runs to determine an optimized correction parameter. 19.The system of claim 18, wherein performing the statistical analysis bythe one or more processors includes: for each run of the multitude ofruns, determining a first median of the correction parameters for eachof a closest set of runs backward in time and determining a secondmedian of the correction parameters for each of a closest set of runsforward in time; and selecting either the first median or the secondmedian as the optimized correction parameter based on which of the firstor second median is closest to the correction parameter determined forthat run.
 20. A non-transitory computer-readable storage medium on whichcomputer readable instructions of a program are stored, theinstructions, when executed by one or more processors, cause the one ormore processors to perform a method for estimating orientation ofcameras mounted on vehicles, the method comprising: determining aninitial pose estimate of a vehicle having a camera mounted thereon, theinitial pose estimate being a position and orientation of the vehiclerelative to the world; processing image data from images captured by thecamera to obtain one or more tracks, each track including a sequence ofmatched feature points stemming from a single three-dimensionallocation; and determining from the initial pose estimate and the tracks,a correction parameter to correct orientations of the images captured bythe camera, wherein determining the correction parameter includesevaluating a set of vehicle poses, three-dimensional locations from theone or more tracks, and an orientation of the camera relative to one ormore sensors of the vehicle.